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improve student outcomes. In the words of district, parent, 
and community shareholders, “This provides a supportive, 
consistent process to make change; from the families’ 
point of view it gives a consistent message ... so they 
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states and how Kentucky is supporting its Cohort II regions, 


districts, schools, teachers, and families. 
From one cohort to the next, the lessons learned are used 


to benefit current and future states. http://sisep.fpg.unc.edu/ 


About the State Implementation and Scaling 
up of Evidence Based Practices (SISEP) 
Center The SISEP Center is a project of the National 
Implementation Research Network (NIRN) and is funded 
by the U.S. Department of Education's Office of Special 
Education Programs (OSEP). The purpose of the SISEP 
Center is to help establish implementation and scaling 
capacity in state, regional, and district educational 
systems. SISEP provides content and technical assistance 
toward establishing large-scale, sustainable, high-fidelity 
implementation of effective education practices. Our shared 
goal is to maximize academic and social outcomes of all 
students, especially students with disabilities. 


The SISEP Center also provides technical assistance to 
increase knowledge of evidence-based implementation 
supports for evidence-based practices in Charter 
Management Organizations, Local Education Agency 
Superintendents and other leaders, and Technical 
Assistance Centers and projects funded by the U.S. 
Department of Education’s OSEP. 
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Kentucky's educators are creating systemic change 

to produce noticeable improvements in learning for all 
students. Systemic change has vexed education systems 
for decades (Tyack & Cuban, 1999). In recent years, it has 
become clear that implementation science is the missing 
ingredient in efforts to purposefully improve student 
outcomes in state systems (Fixsen, Blase, Metz, & Van 
Dyke, 2013; Fixsen, Blase, Ward, Jackson, & Coffey, 

in preparation). The Kentucky Department of Education 
(KDE) has taken a thoughtful and persistent approach 

to systemic change in partnership with its educators 

and communities. Kentucky has documented the initial 
changes in the education system and the benefits to 
system alignment and cohesion for improved outcomes. 
We summarize a few of the key actions of Kentucky's 
educators here, as well as some of the results of their 
efforts so far. In August of 2014, with the intensive support 
of the State Implementation and Scaling up of Evidence 
Based Practices (SISEP) Center, Kentucky began to 
develop its implementation infrastructure and capacity, 
with goals for student learning in the area of math, as 
they developed the knowledge and skills to use the Active 
Implementation Frameworks (See Appendix A: Fixsen, 
Naoom, Blase, Friedman, & Wallace, 2005). 
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IDENTIFYING AN AREA OF FOCUS TO DEVELOP AN 
IMPLEMENTATION INFRASTRUCTURE 


Designing an implementation infrastructure and refining 
implementation efforts requires an area of focus. The 
heart of the Kentucky Department of Education’s 

Active Implementation work is the State Systemic 
Improvement Plan (SSIP). The SSIP fosters use of a 
Learning Laboratory where Implementation Teams are 
engaging in trial and learning. Specifically, their State 
Identified Measurable Result (SiMR) goal is to improve 
math outcomes for students in the Commonwealth by 
“increasing the percentage of students with disabilities 
performing at or above proficient in middle school math, 
Specifically at the 8th grade level, with emphasis on 
reducing novice performance, by providing professional 
learning, technical assistance and support to elementary 
and middle school teachers 
around implementing, scaling 
and sustaining evidence- 
based practices in math.” 
The federal SSIP mandate 
provides support for the 
scaling work in Kentucky and 
other states. 


The implementation infrastructure is the foundation for 
effective use of any practice through organizational 
alignment and use of three types of Implementation 
Drivers (Fixsen, Blase, Naoom, & Wallace, 2009; 
Bertram, Blase, & Fixsen, 2015; Metz & Bartley, 2012). 
For the purpose of a common term, innovation will be 
used in this paper to describe any new practice, program, 
or intervention used by teachers in the classroom. 
Competency Drivers focus on selection, training, 
coaching, and use of a fidelity measure to develop the 
confidence and skill of staff to use a practice effectively. 
Organization Drivers make use of a comprehensive 
data system to resolve barriers to implementation at the 
practice and organizational level. Leadership Drivers 
guide the resolution of adaptive and technical barriers 
at all levels of the education system, from KDE to the 
region, district, school, and classroom (Heifetz & Laurie, 
1997). 


Implementation Drivers 


The implementation infrastructure is 
the building block to support effective 
practice through organizational 
alignment and use of the 
Implementation Drivers (NIRN, 2013). 
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LEADING SYSTEMIC CHANGE: THE CRITICAL ROLE OF THE STATE 
TRANSFORMATION SPECIALIST AND THE STATE MANAGEMENT TEAM 


Developing implementation capacity (knowledge, 

skills, abilities) is reliant upon the State Transformation 
Specialists’ intentional and continuous development of 
the knowledge and skills to use implementation research 
and practices themselves, and in the first cohort of 
linked Implementation Teams: teacher, school, district, 
region, and the state. Linked Implementation Teams 
(See Figure1) define, develop, and monitor the use of an 
implementation infrastructure by districts and their schools 
to help assure dramatically and consistently improved 
student outcomes. 


Figure 1 
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A Cascading System of Supports for Systemic Change 


Align, integrate, leverage system 


in a Transformation Zone. 


Teachers Students 


and organization supports for 
teachers and student learning 


_ FL_ 
mca, | XJ 
F Team 


Teachers’ use of effective 
Disticunegion | ff practices; With fidelity and 
Implementation consistently good outcomes 
F Team 
State/Federal E | Active 
Implementation Implementation 
Team Frameworks 


Two State Transformation Specialists are leading the 
state’s Active Scaling work in partnership with SISEP and 
KDE executive leadership (known in this paper as the 
State Management Team), implementation workgroups, 
Regional, District and School Implementation Teams, 
teachers and community shareholders. Together, they 
develop the readiness and capacity of individuals 

and teams to use the best available evidence from 
implementation science research and practice. State 
Transformation Specialists help to assure the success 
of the overall implementation infrastructure and scaling- 
up capacity in the state. They initiate and serve as core 
members of the State Management Team and regularly 
use data to communicate successes and barriers to 
solve. They ensure that Practice to Policy and Policy 

to Practice Feedback Cycles are used to leverage 
existing resources, reduce waste, and create alignment 
and cohesion from the classroom to the state — all to 
support continuous improvement of teacher practice 
and improved student outcomes. Accountability rolls 
uphill to the Implementation Team with the authority to 
remove barriers in districts, schools, and classrooms. The 
onus is no longer on the individual teacher to improve 
outcomes; the onus is on implementation teams who 
take responsibility for effective use of practices in the 
classroom. 


“It is helpful when SISEP and State 
Transformation Specialists ‘lean in’ 


to help us improve our processes.” 


—— Implementation Team Member 
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State, regional, and district executive leadership 

commit to meeting every month to analyze capacity 

and implementation data in support of implementation 
teams and fidelity of teacher practice to improve student 
outcomes. There are three main aspects of capacity 
measurement, which provide information about the 
systems, activities, and resources required to initiate 
and sustain improved student outcomes by teams at 
every level of the system. Fidelity measures the extent 
an innovation is used as intended by teachers in the 
classroom. Outcome refers to both interim and summative 
student data. In time, teams at every level of the system 
are in place, collecting and using capacity, fidelity, and 
outcome data for action planning and monitoring goal 
attainment. /mplementation data are used to measure 
the systems of support (training, coaching, data use) to 
support effective teacher practice. 


State Transformation Specialists become experts 

in coaching implementation teams to use the 
Implementation Stages, a non-linear process to 
Explore and mutually agree to /nstall teams and the 
necessary supports for effective use of a practice. 


Then, they provide the intensive support needed during 
Initial Implementation when teachers begin to use 

a new practice. Over time, often three to five years 

under optimal conditions, organizations attain Full 
Implementation as effective teacher practice produces 
notable improvement in student outcomes. The first 

group of Kentucky's educators who mutually agreed to 
participate are dedicated to developing and sustaining 
linked Implementation Teams in a Cascading System of 
Supports from the state to the region, district and school— 
all in support of effective teacher practice (Blase, Fixsen, & 
Ryan Jackson, 2015) (Figure 1). Systems cannot change 
all at once (Blase, Fixsen, Sims, & Ward, 2015). Instead, 
change is initiated and the intended and unintended 
outcomes of change are managed in a Transformation 
Zone (Fixsen, Blase, & Van Dyke, 2012). In this paper, 

we describe how Kentucky's first Transformation Zone is 
leading systemic change. This first Transformation Zone is 
comprised of two Regional Education Agencies (known in 
Kentucky as Cooperatives), one very large district that is a 
Cooperative (represented in the district data in this paper), 
5 districts (large urban to small rural), 7 schools, and 44 
teachers. 


Capacity measurement refers 


to the systems, activities, 
and resources required to 
initiate and sustain improved 


education outcomes. 


Implementation Stages 


Exploration, Installation, Initial 

and Full Implementation outline 

the integrated, non-linear process 

of deciding to use an effective 
innovation and finally having it fully 
in place realizing promised outcomes 
(NIRN, 2013). 
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THE PIVOTAL ROLE OF THE REGIONAL EDUCATION AGENCY 


Another critical role is that of the Regional Educational 
Agency. Regional Implementation Teams are the link 
between the state and its districts, schools, and teachers. 
Regional Implementation Teams take responsibility for 
the development of implementation capacity in their 
districts with the support of the State Transformation 
Specialists and SISEP. They leverage improvement 
efforts (Fixsen, Blase, Ward, & Sims, 2014). As liaisons, 
the Regional Implementation Teams meet monthly with 
district and school leadership. Teachers and school staff 
report barriers to effective practice, so the District, Region 
or State Management Team can remove them through 
rapid response. Implementation Teams, at every level of 
the system, take responsibility for implementation as they 
select, install, and sustain practices that are responsive to 
teacher and student need. When teachers see barriers 
removed in support of their effective practice, they feel 
empowered. Principals and teachers no longer feel 
burdened to “do it all.” Instead, they create a shared vision 
of collective commitment and accountability. 


Figure 2 
Transformation Zone 
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Two Regional Cooperatives and one of the larger 
districts in the United States agreed to participate in 
the first Transformation Zone: Ohio Valley Educational 
Cooperative (OVEC), Southeast/Southcentral 
Educational Cooperative (SESC), and Jefferson 
County Public Schools (JCPS). OVEC serves over 
55,000 students in 14 districts, in 12 counties in north 
central Kentucky. SESC serves 81,000 students in 27 
districts, in 16 counties in southeast and southcentral 
Kentucky. Jefferson County Public Schools serve over 
100,000 students in 172 schools. (See Figure 2.) The 
first step is the careful selection and design of the initial 
Transformation Zone. 


A Transformation Zone is a representative portion of the 
state system (urban and rural districts with varied size 
and demographics), typically comprised of two to three 
geographic regions. In each region, initial transformation 
work is focused on one to three districts and one to 
three schools in each district (depending on the size of 
the district). A Transformation Zone is small enough for 
Implementation Teams to be established at every level 
of the system and to develop the capacity of each team 
(knowledge and skills to use implementation science 
research and practice) and thus become functional. Yet, it 
is also large enough to “disturb the system” and uncover 
long-standing barriers to effective use of practices 
designed to improve student learning. 
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Implementation Teams act as “sensors” who identify 
successes and barriers based on the valuable feedback 
from families, teachers, schools, districts, and regions. In 
turn, successes are replicated and barriers are lifted to 
the team that has the authority and resources to sustain 
success and surmount barriers. Resolution of barriers 
requires intentional use of data for problem solving, 
solution development, and action planning that is 
monitored for success using Improvement Cycles 
(Plan-Do-Study-Act; PDSA) to support intentional, 
systematic change. In time, (usually 3-5 years) the system 
can be replicated in the second Transformation Zone, 
more quickly, and with a high degree of probability that the 
systems and practices will be generalized for use in any 
region, district, and school. 


Improvement Cycles 


Plan-Do-Study-Act Cycles support 
systematic and intentional change 
(NIRN, 2013). 


Using SISEP'’s “just enough, just in time” approach, 
SISEP, State Transformation Specialists and Regional 
Implementation Teams began to explore the readiness 
and capacity of districts and their schools to participate 

in the first Transformation Zone in March of 2015. Using 
an active modeling approach (“I do, We do, You do”) 

to teaching implementation skills, SISEP and the State 
Transformation Specialists modeled the use of Active 
Implementation activities (e.g., selection of team members 
and expectations regarding roles and responsibilities). 
Through the intentional use of iterative PDSA Cycles, 
regional members, State Transformation Specialists, 

and SISEP continued to provide coaching and 
implementation informed support for improvement 
purposes. Regional Implementation Teams also gathered 
four times per year for a full day to learn from one another's 
success and challenges, and they expressed the value of 
coming together. 


“The quarterly all day Transformation 
Zone meetings helped us learn how 
the other regions were developing 
District and Building Implementation 


Teams so we could replicate the 
process.” 


—— Regional Implementation Team member 
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Using the best evidence from implementation science to 
ensure teachers and school staff have what they need when 
they need it creates a belief in school staff that the focus 

is on a system of support for effective use of a practice 

(and not on individual teacher evaluation). Implementation 
Teams are persistent in their communication; the onus is 

no longer on the teacher and principal, but on the systems 
of support each teacher and principal receive to effectively 
use a practice to positively impact student outcomes. 
Shareholders in Kentucky acknowledged, “This is a big shift 
in our thinking and our culture,” as schools focused on trial 
and learning to design and refine implementation efforts 

in the first Transformation Zone. The aim was to hone the 
process so it could be scaled up successfully, with good 
outcomes, in the second Transformation Zone. 


“This is a big shift in our thinking 
and our culture.” 


— Kentucky Shareholder 
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ENGAGING KENTUCKY SHAREHOLDERS 


Simultaneously, as Implementation Teams 


Usable Innovation were formed at every level of the system, the 
An innovation needs to be state provided the resources for a diverse 
teachable, learnable, doable, and team of Kentucky shareholders to develop a 
readily assessed in practice if it is usable math innovation. A Usable Innovation 
to be used effectively and benefit is teachable, learnable, doable, and easily 

all students (NIRN, 2013). assessed in practice (NIRN, 2013). To begin 


the process of developing a usable math 
innovation, the state assembled a team, 
known as the Instructional Practices and 
Content Team (IPAC), with members who 
were carefully selected for their knowledge 
and skills in math content, implementation 


practices, and implementation capacity in 
regional, district, and school contexts where 
math innovations would be used. This diverse 
team was responsible for the development of 
the Practice Profile. For a complete description 
of how Kentucky developed a Practice Profile, 
see Kentucky’s Usable Innovation Development 
(in preparation). Using state-provided 
resources, the diverse team co-created the 
Practice Profile so all Kentucky shareholders 
could see themselves in the process and 

final product. The complex procedure took six 
months to complete and it was accomplished 
through strong partnerships with Kentucky 
shareholders: state, regional, district, school 
staff, instructional coaches, university and 
community partners, and purveyors of 
innovations used in Kentucky's schools. 

This purposeful and intentional process 

was designed and owned by Kentucky’s 
shareholders and educators. It ensured a 
quality standard for math practices 

in Kentucky's classrooms. 
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Figure 3 
Kentucky's Usable Math Innovation Process 


WHO: INSTRUCTION PRACTICES AND CONTENT TEAM (N=20) 


Diverse team with knowledge and expertise in mathematics and implementation 
Higher Education ~ Regional, District, School Partners ~ State Mathematics Organizations 


HOW 


Gathered and analyzed research on 


Surveyed educators to gather math best practice 


information on: 


¢ What and How math innovations 
were used in districts and schools 
Evaluated widely used math innovations 
(N=23) for: need, fit, evidence, readiness, 
capacity, resources 


e KY Educators’ philosophy for 
teaching math 


WHO: COMPETENCY TEAMS 


Workgroups with expertise in training (N=9), coaching (N=20), data use (N=18) 


HOW 


Develop a training system of support and 
assessments to monitor and strengthen 
training effectiveness 
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assessments to monitor and strengthen 
coaching effectiveness 


HOW 


Develop a data system to monitor and strengthen training and coaching 


systems of support that result in high fidelity of any innovation 
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Of equal importance were the workgroups that designed 
a comprehensive training, coaching, and data system 

to support effective use of the math Practice Profile by 
coaches, principals, and teachers. Once the Practice 
Profile was complete, some IPAC team members joined 
one of three workgroups responsible for developing the 
training, coaching, data systems, and measures. Experts 
in the state who were providing coaching and training on 
the use of math practices and data use were also mutually 
selected to participate in this worthy process. For a 
complete description of the timeline, tools, and measures, 
see How Kentucky Shareholders put into practice a 
Usable Innovation: Measuring the Independent Variable 
(in preparation). As Implementation Teams at every level 
of the system became functional, they collected and used 
capacity data to develop a sustainable implementation 
infrastructure that would eventually apply to any program, 
practice, or initiative. 
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USING CAPACITY AND FIDELITY DATA IN KENTUCKY’S 


FIRST TRANSFORMATION ZONE 


Measuring implementation capacity at every level of 

the system for full and effective use of a practice that 
benefits all students is critical to alignment and cohesion 
of implementation efforts. Capacity is measured using the 
State Capacity Assessment (SCA: Fixsen, Ward, Duda, 
Horner, & Blase, 2015), Regional Capacity Assessment 
(RCA: St. Martin, Ward, Fixsen, Harms, & Russell, 2015), 
District Capacity Assessment (DCA: Ward et al., 2015), 
and the school-level Drivers Best Practice Assessment 
(DBPA: Fixsen et al., 2015). Fidelity is measured using the 
Observation Tool for Instructional Supports and Systems 
(OTISS: Fixsen, Ward, Ryan Jackson, & Chaparro, 2016). 
Capacity development in Kentucky's first Transformation 
Zone took 40 months, as of January 2018. 


Figure 4 
Kentucky Capacity Assessment Scores 


Kentucky Capacity Assessment Scores: 
State (SCA), Region (RCA), District (DCA), 
School (DBPA) and Fidelity (OTISS) 
First 40 Months of Active Implementation 
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Aggregate data in Figure 4 represents the capacity data 
score (e.g., total percentage) for the State Management 
Team, two regions, 5 districts, and 7 schools with fidelity 
data from 49 teachers. See Appendix B for individual 
capacity assessment and OTISS graphs; SCA, RCA, DCA, 
DBPA and OTISS. Note, one district exited the initiative in 
2017 and not all teams completed the capacity assessment 
every six months as suggested for best practice. 
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clear to me just what we need to be 


doing.” 


— Reflection at a Capacity Assessment 


Note. The Cascading System 

of Supports requires the state to 
complete the SCA first, then the 
region completes the RCA, then 
the district completes the DCA. 
The DBPA and fidelity data are 
often collected simultaneously. Best 
practice is to collect capacity data 
every six months. Some teams did 
not collect capacity data every six 
months based on team readiness. 
Fidelity data is collected at least 
three times per year to coincide 
with collection of math screening 
data. Some districts collect fidelity 
data more frequently. One district 
exited the project in 2017. See 
Appendix B for individual graphs. 
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“We-use the capacity data to develop 
our action plan and our action plan 
helps us be much more focused. With 
selection [the Selection Driver], we did 
not have a job description for a math 
teacher; the district did but we did 


not, so we looked at the district job 
description and added the math 
Practice Profile components. We used 
the capacity assessment questions to 
develop interview questions for math 
teachers and now we use them.” 


— Building Implementation Team members 


Initial results suggest that the inputs at one level are 

the outputs at the next level. The relationship between 
systemic change at the state and regional level align with 
development of functional implementation teams at 

the district and school, creating a Cascading System 

of Supports for teachers' effective use of their district's 
chosen math innovation (or evidence of improved 
scores on the OTISS fidelity measure). Implementation 
Teams are encouraged to complete the capacity 
assessment every six months to develop action plans, 
and monitor progress regularly for best practice. As linked 
Implementation Teams establish capacity, the cascading 
system of supports becomes more effective and efficient 
as implementation skills and communication between 
teams strengthen and expand. The graph in Figure 4 
demonstrates that the purposeful use of implementation 
research and practice creates systemic change from 

the state to the school. For an in-depth description 

of capacity assessment data and how teams were 
developed, supported, and linked see Trial and Learning 
in Kentucky's Transformation Zone (in preparation). The 
data suggest that it takes time to establish Readiness— 
a developmental point—and collective commitment to 
accountability as Implementation Teams are established 
and take responsibility for developing readiness for 
change. Readiness is not a pre-existing condition or 

an enduring characteristic of a person, organization, or 
system (Fixsen, Blase, Horner, Sims, & Sugai, 2013). 


Readiness is a developmental 
point. Readiness is not a 
pre-existing condition or an 
enduring characteristic of 

a person, organization, or 
system (SISEP, 2013). 
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STATE CAPACITY ASSESSMENT 
Specifically, at the third administration of the SCA (64%, 
18th month, March 2016) the Kentucky Department of 


Education met the SCA end of year one goal (EOY 1, “We should have been asking these 
60% total score) as they clarified roles (e.g., liaisons questions all along.” 

between levels — state to region, region to district, etc.), 

how teams would function (e.g., linked communication — Reflection by a team member at a 
protocols to feed barriers and successes up and down Capacity Assessment 


the system), and organizational structures (e.g., service 
delivery models for effective use of training, coaching, 
and data). 


REGIONAL CAPACITY ASSESSMENT 

The aggregate data indicate the two Regional 
Implementation Teams met EOY 1 goal of 60% in an 
average of thirteen and a half months with a mean 
systematic and transparent score of 65% (or 61% and 68%), four and a half months 
communication up and down sooner than the state. As the State Management Team 
the system; soliciting feedback became functional, it provided an enabling context for 


An Enabling Context is 


and using data at the practice the work of the Regional Implementation Teams through 
level to inform how the system the use of systemic and transparent communication to 
supports effective use of a quickly resolve barriers to implementation in regions 
practice to positively influence and their districts (Elmore, 2002; NIRN, 2017). These 
the policy, regulatory, and regions, with the support of the State Transformation 
funding environments at the Specialists, the State Management Team, and SISEP 
state level (NIRN, 2017). began to develop the capacity of the five districts that 


mutually agreed to participate in the first Transformation 
Zone in the spring and summer of 2015. 
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DISTRICT CAPACITY ASSESSMENT 

As the state and the first Transformation Zone regions 
began to learn about the strengths and challenges to 
effectively use math innovations in districts' schools, the 
learning laboratory began to reveal strengths (e.g., active 
engagement and commitment of district executive leaders) 
and barriers (e.g., absence of effective training and 
coaching supports for teachers). The aggregate District 
Capacity Assessment (DCA) data in Figure 4 represents 
data from the five Transformation Zone districts. The data 
demonstrate it took twenty-four months for districts to 
meet EOY 1 goal of 60% (fifth administration or 74%), 

an average increase of 47% from baseline. Note, from the 
first to second administration scores remained almost flat 
(3% increase), suggesting it took time for districts to 
establish high-functioning teams, readiness, and capacity 
to use the best evidence from implementation research. 


Simultaneously, as Implementation Teams were formed 
from 2015-2016, the state workgroups were developing 
and /nstalling training, coaching, and data systems for 
use in districts and their schools. Then Implementation 
Teams, at all levels of the system, were trained on their 
use. Districts used capacity data to action plan and report 
barriers to their Regional Implementation Team and 

State Transformation Specialists, so the implementation 
infrastructure and current systems could be strengthened 
and aligned. 


“The fault cannot lie in the one 
responsible for the repair.” 


— Ross Ashby, psychiatrist and a 
pioneer in the study of the science 
of communications and systems 
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SCHOOL CAPACITY ASSESSMENT 

In the spring of 2016, with the support of their region, 
State Transformation Specialists, and SISEP, districts 
mutually selected the first schools to participate in 

the Transformation Zone. In the fall of 2016, District 
executive leadership began to attend Building 
Implementation Team meetings and collect school 

level capacity data using the Drivers Best Practice 
Assessment (DBPA). The school level capacity data 
(DBPA) suggest that, once the state, region, and 
district developed the implementation infrastructure and 
capacity to use the Active Implementation Frameworks, 
the school's ability to progress at an accelerated rate 
was realized. The data from seven schools indicate 
that schools met EOY 1 goal for the DBPA in twelve 
months. Baseline aggregate score was 26% at the first 
administration and 67% at the second, an average 
increase of 41%. 


“We are much more focused and | 
would like to say | feel so fortunate 
to be part of this work. | know we 

are going to build teachers’ capacity, 
but not only are we going to build it, 
we are going to be able to sustain it.” 


— School principal in the Transformation 
Zone 


At State Implementation and Scaling-up 
of Evidence-based Practices 


y N | R N NATIONAL IMPLEMENTATION 
RESEARCH NETWORK 


FIDELITY ASSESSMENT 

As Building Implementation Teams began to collect and 
use school level capacity data, they were simultaneously 
collecting baseline fidelity data using the Observation 
Tool for Instructional Systems of Support (OTISS) 
measure. The OTISS was developed based on Hattie’s 
(2009) meta-analysis of over 800 meta-analyses 
including over 50,000 studies related to student 
achievement. Items on the OTISS are scored based on 
direct observation of teacher instruction during a 10- 
minute classroom walk through by a trained observer. 
The OTISS is scored on a 3-point scale; 2, fully 
observed; 1, partially observed; 0, not observed. 
Ascore of NAis also provided—no opportunity to 
observe. The OTISS is designed to observe teacher 
instruction and evaluate the implementation supports 
provided to teachers — it is not a teacher evaluation tool. 


At a recent SISEP Forum, a state 
leader asked a teacher whether there 
is anything you can point to that 
communicated the OTISS is not an 
evaluation. 


“Well,” the teacher replied, “every 
time we met they kept bringing it up... 
‘It’s not about you as an individual 


teacher.’ So, constantly hearing that. 
Every meeting was very positive and 
organized, everything was for our 
purpose.” 


A Regional Implementation Team 
member added, “We had a pre- 
meeting before we ever went into 
the classroom with all teachers. We 
showed them the OTISS and told 
them it wasn’t about them: it was 
about the system. And we showed 
them the data... that there weren't 
going to be any names on it. They 
knew what we were looking for and 
what we were going to do with the 
data.” 
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“You really meant it is the 
system and not me.” 


— Teacher participating in the SISEP 
Active States Forum 


SISEP suggests that schools collect OTISS data 
at least three times per year (beginning, middle, 
end). It is important to note that the number of 
months between collection of OTISS data varied 
between schools. The fidelity data from six schools 
implies that teachers were able to meet, within one 
percentage point, the OTISS goal of 80% from the 
first observations (December of 2016) to the fourth 
observations (October 2017). Scores dipped to 
68% at the third observation, rebounding to 

79% at the fourth observation. The data suggest 
the OTISS is sensitive to change. Scores may dip 
as observers strengthen their beliefs regarding 
observed teacher behaviors that constitute a score 
of two. Scores can also dip with staff turnover 

as initial and ongoing training and coaching are 
provided based on OTISS scores. As Building 
Implementation Teams reviewed capacity and 
aggregate fidelity data, they identified barriers at 
the school level that the school did not have the 
resources to solve. 


As barriers at the school level were unveiled, 
Implementation Teams reported barriers to the team 

with the authority and resources to remove the barrier. 
Teams, at every level of the system, created an Enabling 
Context for effective use of math innovations by teachers 
and school staff. Currently (2017-18 school year), 
implementation teams in the Transformation Zone are 

in Initial Implementation and using capacity, fidelity, and 
implementation data (training and coaching) housed in 
Kentucky’s Data Dashboard. Teams are solving systemic 
barriers, strengthening the implementation infrastructure 
to support effective practice, and meeting school and 
district goals for improved math outcomes. The capacity 
and fidelity data typically predict improved outcomes; 
when capacity increases at each level of the system, then 
innovation fidelity and student math outcomes also should 
increase. 


“Lam looking forward to this process 
because we get to say, ‘Hey, what is 
the next best question we can ask?’— 


and try to solve it.” 


— Team reflection of how the Data 
Dashboard is used 
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IMPROVED STUDENT OUTCOMES 


Preliminary data from one of the three districts with This careful and intentional process increases the 
repeated capacity, fidelity, and proximal student data probability that systems change is teachable, learnable, 
suggests if implementation supports are provided doable, and easily assessed in practice to improve 


intentionally and with expertise (the independent variable), | Outcomes for teachers and their students. 
then teachers and school staff will use effective innovations 
consistently with fidelity and produce improved student 
outcomes (the dependent variable). In one Transformation 


“ . P ATF 
Zone district (two schools and 15 teachers), the percentage Implementation with fidelity makes 


of students meeting benchmark in math at the elementary change much more sustainable.... We 
and middle school level increased 29.5% during the 2017- are beginning to get data linking 

18 school year: from 21.5% (fall 2017) to 38.5% (winter outcomes to use of math instruction 
2018) to 51% (spring 2018), based on the use of a math with fidelity.” 

screening measure. The number of students scoring 

below benchmark decreased by 11.5% and students — Teachers and staff, discussing OTISS 
scoring well below benchmark decreased 18%. While data 


this is a very small sample, it is preliminary evidence that 
suggests if implementation supports are provided, then 
schools can produce improved student outcomes. If not, 
teams use capacity, fidelity, and implementation data (e.g., 
data from training and coaching) to ask why not? The 
Learning Laboratory in Kentucky's first Transformation 
Zone is developing and testing the roles, functions, and 
implementation structures that can be scaled-up at an 
accelerated rate in the second Transformation Zone 
(installed in the winter of 2018). 
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CONCLUSION 


State Transformation Specialists, Regional Implementation 
Teams, and SISEP provided support for District and 
Building Implementation Teams at monthly meetings during 
a time when there were leadership and staffing changes 

in Kentucky. New roles, functions, structures, and linkages 
in a new system of Implementation Teams typically are 
fragile as they are formed and developed. On-boarding new 
members and continued capacity development of teams, at 
every level, during times of high turnover emphasizes a key 
role of SISEP and the State Transformation Specialists as a 
source of stability during potentially destabilizing events. 


“We'can readily see how this 
work can apply to other initiatives. 


— Kentucky educator, discussing 
active implementation 


Equally important was the unwavering commitment 

and support of the members of the State Management 
Team. They were never distracted in their attention to the 
development of a state implementation infrastructure that 
could be replicated for use with any initiative, innovation, 
or mandate by their shareholders. They ensured 
workgroups were representative of diverse shareholders 
from every level of the system to establish the training, 
coaching, and data systems required to support teachers’ 
effective use of math innovations in the classroom. 

The communication was clear and consistent: the 

onus was not the sole responsibility of teachers and 
school staff; it was on the provision and use of an 
implementation infrastructure to provide the systems of 
support—Competency, Organization, and Leadership— 
for teachers to use, with a high degree of fidelity, the 
district's chosen math innovation. The careful attention 
Kentucky paid to ensuring teams were representative of 
Kentucky's shareholders in turn allowed educators to see 
themselves, their philosophy, and their perspectives in 
the tools and resources—and this created ownership and 
widespread buy-in for their use. Kentucky was intentional 
and purposeful in the development of a sustainable 
implementation infrastructure. They were “making it 
happen.” 
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SISEP's “just enough, just in time” strategies provide ample 
opportunities for teaching and learning at one level while 
simultaneously coaching the use of newly learned skills to 
develop capacity at the next level. The interconnections 
between the schools, district, region, and state take time to 
develop. As indicated in the capacity data from the SCA, 
RCA, DCA, and DBPA, the skills being taught, learned, and 
used at each level are tracked. The Active Implementation 
Frameworks and systems of support are used by each 
Implementation Team; “common language, common 
knowledge, common practice, and common skills” pervade 
each level. These implementation-informed approaches 

to problems and solutions promote collaboration and 
alignment of activities and outcomes within the state 
education system and, ultimately, outcomes that support 
schools, teachers, and student learning. Educators see that 
using the Active Implementation Frameworks is not merely 
another initiative, but a process that can accompany any 
program, practice, initiative, or mandate and spur effective 
use or full implementation for the benefit of all children and 
students. Proficiency in the teams’ use of the core Active 
Implementation Frameworks and the ease of 


communication among the levels has provided an Enabling 
Context in Kentucky's first Transformation Zone as barriers 
are reported to the next level and solved. 


In this way, practice can inform policy development, and 
policy can enhance effective practice in regions, districts, 
and schools. The result is a nimble and responsive system 
of supports for teachers and student learning. 


“From a statewide perspective, 
everything we do has to be 
sustainable and scalable - has to 

be. What we needed to do here was 
rapid development, we needed to 
incubate quite a bit, and fail forward 
with some ideas...we collect the 

fire out of data, but | don’t think we 
effectively use data all the time, that is 
why this is so exciting to me, buckets 
of smart people with a hierarchal 
view — school, district, region, state.” 


mmm C11(-18 B)(e]i-1@)ii(ex-1N 4B) a 


Actively using the research evidence from implementation 
science is complex and challenging work for 
mplementation Teams. Excitement is experienced every 
time a tipping point is reached and the excitement is 

often followed by the next adaptive challenge. Most often 
this process repeats itself for several years as readiness 
or the next right step is established. Then one day, the 
adaptive challenges become smaller, not so complex, and 
less frequent as the system aligns and takes another step 
towards producing educationally significant outcomes. 

he result is a sustainable permanent source of support 


jor continually supporting effective teacher practice and 
improving student outcomes for every child and student. 
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APPENDIX A 


Active Implementation Frameworks 


Usable Innovations are effective and well-operationalized. Well-operationalized 
innovations can be taught and coached so educators can use them as intended 

(with fidelity). An innovation needs to be teachable, learnable, doable, and readily 
assessed in practice if it is to be used effectively to reach all students who could benefit. 


Implementation Teams support the full, effective, and sustained use of effective 
instruction and behavior methods. Linked Implementation Teams define an 
infrastructure to help assure dramatically and consistently improved student outcomes. 


Implementation Drivers are the key components of capacity that enable the success | (4 » 
of innovations in practice. Implementation Drivers assure development of relevant 
competencies, necessary organization supports, and engaged leadership. 


Implementation Stages outline the integrated, non-linear process of deciding to 
use an effective innovation and finally having it fully in place to realize the promised 
outcomes. Active implementation stages are Exploration, Installation, Initial 
Implementation and Full Implementation. 


Improvement Cycles support systematic and intentional change. Improvement 
Cycles are based on the Plan, Do, Study, Act (PDSA) process for rapidly changing 
methods, usability testing for changing innovations and organization supports, and 
practice-policy communication cycles for changing systems to enable continual 
improvement in impact and efficiency. 
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APPENDIX B 


Capacity Assessment and Observation Tool for 
Instructional Supports and Systems Graphs 


The Cascading System of Supports requires the state to complete the SCA first, then the region completes the RCA, then 
the district completes the DCA. The DBPA and fidelity data are often collected simultaneously. Best practice is to collect 
capacity data every six months. Some teams did not collect capacity data every six months based on team readiness. 
Fidelity data is collected at least three times per year to coincide with collection of math screening data. Some districts 
collect fidelity data more frequently. One district exited the project in 2017. 


State Capacity Assessment (n= 1) 
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APPENDIX B 


Regional Capacity Assessment (n= 2) 
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District Capacity Assessment (n= 5 ) 
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APPENDIX B 


School Drivers Best Practice Assessment (n= 7) 
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